Multi-modal pedestrian detection with misalignment based on modal-wise regression and multi-modal IoU

Multi-modal pedestrian detection, which integrates visible and thermal sensors, has been developed to overcome many limitations of visible-modal such as poor illumination, cluttered background, occlusion. By adopting the combination multiple modalities, we can efficiently detect pedestrians even with visibility. Nevertheless, critical assumption multi-modal detection is that images are perfectly aligned. In general, however, this often becomes invalid in real-world situations. Viewpoints different modal sensors usually different. Then, positions on have disparities. We proposed a faster-RCNN specifically designed handle misalignment between two modalities. The consists region proposal network (RPN) detector. introduce position regressors for both modalities RPN Intersection over union (IoU) one useful metrics object but defined only single-modal image. extend it into IoU evaluate preciseness Our experimental results evaluation demonstrate method comparable performance state-of-the-art methods outperforms them data significant misalignment.